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ABSTRACT 



This report contains the results of an exploratory 
statistical analysis to evaluate the performance 
of the Systematic Error Identification System (SEIS) 
and the Vortex Tracking Program (VTP) , when 
tracking weather systems. 



AN EVALUATION OF THE PERFORMANCE OF A 
NEW STORM TRACKING METHODOLOGY 



1 . Introduction 

Weather forecasts made by the Fleet Numerical Oceanography 

Center (FNOC) are based on a numerical weather prediction model 

called the Naval Operational Global Atmospheric Prediction 

System (NOGAPS) . Until 1983 the only available measures of 

model performance were of a global nature (aggregated over 

all the weather systems monitored), such as means, variances 

and root mean square errors. The operational field forecasters, 

on the other hand, prefer error statistics relevant at the 

synoptic level, i.e., measures pertaining to forecasts of 

individual storms and troughs. Such measures would enable 

these forecasters to provide better subjective forecasts at 

the regional level. In 1982, the Naval Environmental Prediction 

Research Facility (NEPRF) began the development of the Systematic 

Error Identification System (SEIS) ; the primary data reduction 

methodology within SEIS is the Vortex Tracking Program (VTP) . 

In the VTP, an atmospheric low/high pressure system is 

mathematically represented as a generalized six parameter 

elliptic function. The six parameters correspond with the 

primary features of a storm, viz., the amplitude A, R the 

semi-major of the elliptic representation of the storm, z the 

eccentricity or the ratio of the semi-major to the semi-minor, 

a the orientation of the ellipse and X , Y the coordinates 

of the center of the storm. The units of measurement are 

millibars (mb) for A, and degrees with respect to the North 

for a while R, X , Y are measured in terms of a 63x63 FNOC 

o o 

hemispheric grid units. For each valid storm, the VTP 

uses an iterated non-linear least squares scheme to estimate 

A, R, z, a, X , Y q within the sea level pressure field for 

the analysis at time t as well as for the associated 

12, 24, 36, 48 and 60 hour forecasts produced by NOGAPS. The 

iteration scheme requires a set of initial guess values 

for the parameters to produce the estimates for the analysis 

field at time t. These estimates are in turn used as guess 

values to produce the 12 hour forecasts; the 12 hour parameter 
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forecasts are used to generate the 24 hour forecasts and so 
on. The estimated parameter values for the analysis field 
at time t are also used as the first guesses for the analysis 
field at t + 12 hours. The estimates for the analysis field 
are usually referred to as verification values. Corresponding 
to each set of forecasted parameter values there will be a 
verification set obtained using the current (for the forecasted 
time) sea level pressure data. The difference between a 
forecasted value and its verification value is called the 
forecast error. SEIS, thus, provides the capability to track 
individual weather systems (by tracking the movement of the 
elliptic representation) and also a means to measure and 
analyze the tracking errors . 

The modified NOGAPS model has been running on a real 
time basis since mid 1983. During the life cycle of each 
valid storm, twice each day (at noon and at midnight GMT) , 
the elliptic parameter estimates are produced for the analysis 
field and the associated 12, 24, 36, 48 and 60 hour forecast 
fields. References [1], '[2] and [3] discuss the VTP and SEIS 
models in more detail. 

The objective of this project is the exploratory 
statistical analysis of the forecast errors to assess the 
performance of the SEIS/VTP model. Data on 80 storms, 
covering the North Pacific Ocean Basin, observed during the 
period January-May 1984 has been used in this study. The 
results of the analysis are described in the following 
sections. Section 2 contains overall measures of performance 
of SEIS/VTP, primarily summary statistics of forecast errors 
pooled over all the 80 storms. Error statistics pertaining 
to the tracking of individual storms are presented in Section 3. 
Conclusions and topics for further research are discussed in 
Section 4 . 
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2 . Analysis of Forecast Errors 

A forecast error is defined as the difference between 
a forecasted parameter value and its verification value; 
an absolute forecast error is the absolute value of a 
forecast error. For each of the five forecasting periods 
(12, 24, 36, 48 and 60 hours) the forecast and the absolute 
forecast errors were subjected to various statistical 
analyses . Tables 1 and 2 contain the means (X) and standard 
deviations (S) for these errors. 



TABLE 1 

SUMMARY STATISTICS OF FORECAST ERRORS 



Forecast 

Period 


Number 

of 

Samples 


A 


e 


R 


a 


X 0 


1 


X 


s 


X 


S 


X 


s 


X 


s 


X 


S 


X 


12 


487 


-1.53 


5.20 


-0.05 


1.49 


0.21 


1. 27 


4.75 


49.83 


-0.06 


0.77 


0.1 


24 


429 


-2.67 


6.69 


-0.07 


1. 47 


0.19 


1. 38 


4.60 


52.14 


-0.08 


1.01 


0.1 


36 


371 


-4.11 


7.68 


-0.13 


1. 86 


0.27 


1. 73 


4.83 


56.10 


-0.14 


1.22 


-0. 


48 


329 


-5.14 


8.50 


-0.18 


1.84 


0.28 


1. 76 


5.51 


53.95 


-0.11 


1.34 


-0. 


60 


288 


-5.08 


9.61 


-0.30 


2.03 


0.21 


1. 85 


5.10 


50.85 


-0.09 


1.45 


-0. 



TABLE 2 

SUMMARY STATISTICS OF ABSOLUTE FORECAST ERRORS 



Forecast 

Period 


Number 

n-F 






£ 




R 


a 


X 


0 


5 


Ui 

Samples 


X 


S 


X 


s 


X 


s 


X 


s 


X 


s 


X 


12 


487 


3.87 


3.79 


0.85 


1.23 


0.85 


0.98 


28.79 


40.93 


0.50 


0.59 


0.5 


24 


429 


5.56 


4.57 


0.92 


1.15 


1. 00 


0.98 


32.65 


40.89 


0.69 


0.73 


0.7 


36 


371 


6.84 


5.39 


1.07 


1. 52 


1. 19 


1.28 


37.26 


42.18 


0.87 


0.86 


1.0 


48 


329 


7.65 


6.32 


1.07 


1. 51 


1. 19 


1.32 


36.65 


39.92 


0.97 


0.93 


1.2 


60 


288 


8. 38 


6.92 


1.20 


1.67 


1. 20 


1.42 


35.20 


36.99 


1.04 


1. 01 


1.3 
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The following general conclusions appear warranted. 

The NOGAPS forecasting methodology does forecast the parameters 
A, R, e, X q , Y q quite well. With regards to the forecasting 
of the orientation a, although the mean errors are not 
excessive, the standard deviations are somewhat high. In 
many cases the forecast errors are negative indicating a 
negative bias, i.e., the forecasted values tend to be on 
the low side of the verification values. With a few 
exceptions, the means and standard deviations increase with 
an increase in the forecasting period; this is to be expected 
in view of the higher levels of uncertainty involved. 
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The autocorrelations for lags 1 to 5 between the 

forecast errors are presented in Table 3. Except for 

the lag-one autocorrelations of about .30 for the forecast 

errors of A, X , Y the rest of the autocorrelations 
o o 

are quite negligible. This implies that a large error in 
forecasting a parameter at a given time will not have a 
lasting effect on future forecast errors. Also, the 
correlation matrices (correlations between the errors in 
forecasting A, a, R, e, X q , Y ) in Table 4 show that these 
correlations are negligible with one exception — the 
correlation between the errors in forecasting R and e 
is around .5. This may be interpreted to mean (with the 
one exception) that a large forecast error for one parameter 
will not have a detrimental effect on the estimates of the 
other parameters. 

In an attempt to model the statistical behavior, 
gamma distribution were fit to the 12, 24, 36 and 48 hour 
absolute forecast errors. The histograms with the fitted 
gamma distribution superimposed are in Figures 1-18. Gamma 
distributions appear to serve as good statistical models 
of the absolute forecast errors for A and R. In the other 
cases, the lack of fit may be attributed to a higher peakedness 
in the data; a Weibull distribution may provide a better fit. 
Although no graphs are presented, gamma distributions did not 
provide a good fit to the forecast errors (appropriately 
translated/shifted to make them positive) also. Further 
work will be necessary to determine the most appropriate 
statistical distributions to model the probabilistic behavior 
of the forecast errors. Proper statistical modeling of the 
error data could be useful for exploring the development of 
uncertainty contours (confidence regions) for the movements 
of weather systems. 
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AUTOCORRELATIONS BETWEEN THE FORECAST ERRORS 
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3 . Statistics of Individual Weather Systems 

To evaluate the performances of SEIS, in tracking 
individual weather systems, and the NOGAPS model in 
forecasting weather systems, data on 20 storms with at 
least 10 records per storm (i.e., 10 sets of forecasted and 
verification values per storm) were examined. The means (X) 
and standard deviations (S) of the forecast errors for these 
20 storms are in Table 5. 

The trends in the forecast errors are similar to the 
global trends observed in the previous section; with the 
exception of the forecasting of a, the forecast errors are 
very small even at the individual storm level. The iterated 
non-linear least square procedure in VTP requires initial 
guess values for each of the parameters A, e, R, a, X q , Y q ; 
the initial guess for a is always specified as zero. We 
conjecture that this may be the cause of the somewhat erratic 
forecasts of a. A better initial guess, closer to the true 
value, may result in a better forecast of a. The SEIS/VTP 
appears to be exceptionally good in forecasting the center 
of a storm. 

For each of the 20 storms the forecasted values of 
A, X q and Y q were plotted against their respective verification 
values. In several cases, the scatter plots indicated an 
approximate linear relationship between the forecasted and 
verification values. A few of these scatter plots are 
shown in Figures 19-30. A linear regression analysis was, 
therefore, performed with the forecasted value as the 
independent variable and the verification value as the 
dependent variable. The least squares estimates of the 
intercept and slope of the fitted line and also the estimated 
coefficient of correlation (a measure of goodness of the 
fitted line) are in Table 6. 
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TABLE 5 



SUMMARY STATISTICS OF FORECAST ERRORS FOR INDIVIDUAL STORMS 



)rm 

nber 


Number 

of 

Records 




A 




e 


R 


a 


X 

0 


X 

0 


X 


s 


X 


s 


X 


s 


X 


s 


X 


S 


X 


S 


1 


19 


-9.9 


5.2 


0.6 


1. 3 


0. 8 


1.5 


4.4 


99.0 


5.3 


0.8 


0.1 


0.3 


2 


18 


-10.3 


5.1 


0.5 


1.2 


0.7 


1. 5 


4.0 


101. 7 


0.1 


0.8 


-0.1 


0. 3 


3 


34 


-1.2 


4.9 


-0.6 


1.5 


-0.3 


1.9 


9.3 


24.5 


-5.2 


0.7 


-l.i 


1. 4 


4 


18 


-6.4 


5.1 


-0.7 


2.8 


0.5 


2.8 


-3.3 


20.3 


1.1 


1.4 


-0. 8 


1.3 


5 


60 


-5.6 


10.1 


0.1 


0.9 


0.6 


1.4 


-5.0 


35.5 


0.4 


1.1 


-0.5 


1. 8 


6 


19 


-1.8 


3.9 


-0.2 


1.0 


1.4 


1.7 


6.9 


31.3 


-1. 7 


1. 3 


3.2 


1.8 


7 


27 


1. 3 


3.9 


0.1 


0.4 


0.5 


0.9 


34.9 


32.5 


0.1 


0 . 8 


-0.9 


1.4 


8 


41 


-8.0 


9.3 


-1. 3 


3.3 


0. 7 


2.3 


3.5 


47.2 


-0.7 


1.3 


0.7 


1.4 


9 


25 


-3.2 


16.0 


-2.6 


4.0 


-0.5 


1.4 


11.0 


74.8 


-1.1 


1.6 


0.5 


0.8 


0 


26 


4.8 


7. 2 


-0.1 


2.4 


0.4 


1.1 


-13.4 


86.2 


0.8 


1.6 


-0.4 


0.5 


1 


23 


-6.5 


12.8 


0.4 


0.6 


0.7 


1.6 


9.3 


55.5 


0.1 


1.3 


0.0 


0.5 


2 


13 


0.4 


5.5 


1.5 


2.3 


0.2 


0.7 


-10.7 


29.2 


0.1 


0.9 


0.0 


0.5 


3 


63 


-2.4 


8.3 


-0.1 


1.7 


-0.3 


1.1 


-6.9 


54.4 


0.5 


1.1 


0.8 


0.6 


4 


10 


-11.6 


9.6 


-0.1 


0.8 


-0.8 


1.1 


22.1 


66.6 


1.0 


1.0 


-0.2 


0.5 


5 


20 


-0.7 


6.8 


0.9 


1.4 


0.9 


2.1 


-7.3 


36.7 


1.2 


2.0 


-0.9 


2.0 


6 


27 


-2.0 


6.7 


-1.1 


1.2 


-0.9 


2.0 


7.5 


33.0 


-0.1 


1.3 


0.1 


1.2 


7 


13 


-6.6 


4.2 


-0. 3 


0.7 


-0.7 


0.8 


-17.6 


17.2 


0.0 


0.5 


0. 3 


0.5 


8 


95 


-3. 4 


6.7 


-0.2 


1. 1 


0.2 


1.5 


12.9 


52.0 


0.2 


1.1 


0.9 


2.2 


9 


12 


-5.1 


9.4 


0. 3 


1.4 


-0.1 


1 . 1 


-11.0 


54.8 


0.1 


1.0 


0.1 


1.9 


0 


51 


-3.8 


6.5 


-0.5 


2.7 


-1.3 


2.9 


-20.1 


36.8 


-0.4 


0.5 


0.7 


0.7 
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TABLE 6 



ESTIMATED REGRESSION PARAMETERS 



Storm 


A 


X 

0 




Y 

0 




No. 


Intercept 


Slope Correlation 


Intercept 


Slope Correlation 


Intercept 


Slope j< 


Correlati 


i 


212.9 


0. 79 


0.91 


12.1 


0.45 


0.55 


28.8 


0.25 


0.51 


2 


630.2 


0.37 


0.90 


15.5 


0.25 


0.64 


38.9 


- 0.05 


0.45 


3 


984.9 


0. 003 


0.00 


2.7 


0.84 


0.92 


22.9 


0.29 


0 . 3 C 


4 


495.8 


0.50 


0.37 


20.9 


0.09 


0.16 


23.9 


0.09 


0 . 1 (i 


5 


732.7 


0.26 


0.33 


11.3 


0.35 


0.70 


13.1 


0.62 


o . 8 ; 


6 


962.7 


0.04 


0.05 


11 . 3 


0.45 


0.14 


- 7.3 


1.12 


0 . 7 : 


7 


589.3 


0.41 


0.75 


2.0 


0.88 


0.75 


15.5 


0.61 


0.9 


8 


357 . 8 


0.64 


0.64 


7.5 


0.68 


0 . 87 


12.4 


0.57 


0.8 


9 


1066.7 


- 0.08 


0.09 


8.5 


0.63 


0.85 


8.7 


0.69 


0.9 


10 


714.8 


0.28 


0.28 


- 0.7 


0.99 


0.82 


15.1 


0.46 


0.4 


11 


626.6 


0.37 


0.29 


25.9 


- 0.20 


- 0.29 


4.6 


0.88 


0.8 


12 


143.1 


0.86 


0.47 


6.1 


0 . 73 


0.80 


6.8 


0.75 


0.8 


13 


745.0 


0.25 


0.47 


2.0 


0.87 


0.86 


- 0.2 


0.98 


0.9 


14 


- 133.0 


1.14 


0.33 


- 4.0 


1.19 


0.48 


10.7 


0.59 


0.8 


15 


446.2 


0.55 


0.67 


7.9 


0.60 


- 0.0 3 


8.8 


0.75 


- 0.2 


16 


640.4 


0.35 


0.79 


- 1.6 


1.07 


0.80 


- 0.6 


1.01 


0.6 


17 


349.5 


0.65 


0.09 


4.1 


0.81 


0.32 


0.4 


0.97 


0.3 


18 


812.9 


0.18 


0.59 


8.4 


0.52 


0.60 


0.3 


0.99 


0.8 


19 


261.3 


0.74 


0.57 


22.5 


- 0.02 


0.95 


30.1 


- 0.09 


0.9 


20 


959.5 


0.04 


0.81 


10.1 


0.50 


0.91 


27.9 


0.24 


0.9 
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The regression analyses confirmed what was observed 
from the scatter plots, namely, a linear fit in many cases. 

If the functional relationship between the forecasted and 
verification values can be determined with good precision, 
corrective action can be taken to remove this source of 
systematic error in the forecasting scheme. However, the 
results in Table 6 do not lend themselves to the determination 
of the functional relationship. More data needs to be 
examined and different ways of stratifying the data such as 
by geographic regions and/or climatic seasons may prove to 
be profitable. Another possibility is to group the available 
records for a storm according to the forecast periods, i.e., 
all 12 hour forecasts as one group, all 24 hour forecasts as 
another group, etc. Of course, this scheme can only be 
applied to storms with large numbers of records. We tried 
this approach on 4 out of the 20 storms (storms 5, 13, 18 
and 19, each with 60 or more records) . The records for 
each storm were formed into five groups, one for each of 
the forecast periods and a separate regression analysis was 
performed for each group. 

The estimated regression parameters in Table 7 reveal 

a much stronger linear relationship when the data is 

stratified according to the forecast period. Also, one 

can discern a definite pattern in the relationship between 

the forecasted and verification values of the storm's 

amplitude A. For the 12 hour forecasts, the relationship 

is linear with an intercept value of about 260 and slope .7; 

the intercept and slope values for the 48 hour forecasts 

are around 475 and .5 respectively. This is only an 

empirical observation and a more extensive study will be 

necessary to confirm this. Even though there is a strong 

correlation between the forecasted and verification values 

of X and Y , no pattern is evident in the estimates of the 
o o c 

intercepts and slopes of the fitted lines. Another observation 
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that can be made from the correlations in Table 7 is that 
the 12 and 24 hour forecasts, and to a lesser extent the 
36 hour forecasts correspond well with the verification 
values; the efficiency of the forecasting scheme appears to 
drop after the 36 hour forecasts. 
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TABLE 7 



ESTIMATED REGRESSION PARAMETERS FOR INDIVIDUAL FORECAST PERIOD 



in 


forecast 

Period 


A 

[ntercept Slope Correlation 


Intercept 


X 

0 

Slope Correlation 


Intercept 


Y o 

Slope Correlatioj 




' 12 


275.3 


0.72 


0.77 


3.4 


0.81 


0.80 


10.6 


0.68 


0.83 




24 


476.2 


0.52 


0.72 


7.4 


0.56 


0.78 


6.0 


0.83 


0.96 




36 


748.6 


0.24 


0.37 


10.1 


0.40 


0.82 


12.6 


0.64 


0.85 




48 


1062.5 - 


-0.08 


-0.09 


12.7 


0.26 


0.66 


19.0 


0.45 


0.60 




60 


1235.6 - 


-0.25 


-0.24 


13.4 


0.23 


0.72 


22.5 


0.34 


0.61 




12 


454.6 


0.54 


0.55 


-2.0 


1.11 


0.96 


-0.1 


0.99 


0 . 99 




24 


572.5 


0.43 


0.52 


3.3 


0.81 


0.94 


-0.1 


0.98 


0.99 




36 


740.7 


0.26 


0.39 


6.4 


0.63 


0.78 


0.1 


0.97 


0.99 




48 


751.3 


0.25 


0.54 


4.4 


0.78 


0.80 


-1.4 


1.01 


0.99 




60 


762.9 


0.27 


0.74 


-1.0 


0.99 


0.78 


-1.0 


0.99 


0.99 




12 


256.9 


0.74 


0.77 


2.2 


0.90 


0.92 


0.1 


1.00 


0.96 




24 


489.0 


0.50 


0.49 


3.7 


0.81 


0.81 


2.8 


0.93 


0.93 




36 


454.4 


0.54 


0.60 


6.4 


0.67 


0.60 


8.1 


0.78 


0.81 




48 


616.6 


0.37 


0.45 


14.9 


0.25 


0.22 


14 . 8 


0.57 


0.66 




60 


437.2 


0.56 


0.61 


14.3 


0.27 


0.28 


16.8 


0.51 


0.60 


) 


12 


277.6 


0.72 


0.92 


0.8 


0.96 


0.99 


-0.7 


1.02 


0.99 




24 


488.8 


0.51 


0.78 


-0.8 


1.05 


0.99 


-2.3 


1.06 


0.99 




36 


646.1 


0.35 


, 0.54 


-1.6 


1.08 


0.96 


-1.9 


1.06 


0.94 




48 


701.0 


0 . 30 


i 0.50 


-4.3 


1.2C 


l 0.94 


-1.2 


1.04 


0.91 




60 


819.9 


'1 0.1S 


0 . 38 


-3.9 


1.11 


r 0 . 89 


3.5 


0.90 


0.86 
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4 . Conclusions and Recommendations 

This study has demonstrated that the NOGAPS model 
performs exceedingly well in forecasting five of the six 
parameters of the elliptic representation of a storm. 

The maximum mean absolute error in forecasting the amplitude 
A is 8.38 (Table 2) which is less than 1% of the verification 
values that range between 900 and 1,000; the maximum 
standard deviation of these errors in 6.92. Similar positive 
statements apply to the errors in forecasting e, R, X q , Y q 
as can be seen from Tables 1 and 2 and the single high mode 
close to zero in the histograms (Figures 1-18) of absolute 
errors . 

The autocorrelations (Table 3) between the errors in 
successive forecasts of any of the six parameters indicate 
that these errors may, for all practical purposes, be 
treated as independent. Similarly, for each forecast period, 
the errors in forecasting the parameters A, e, R, a, X q , Y q 
appear to be independent (Table 4). What this implies is 
that a large error in forecasting a parameter may not have 
a lasting effect on other forecasts nor will it have a 
carry over effect on forecasting the other parameters. 

Even at the individual storm level, the mean forecast 
errors and their standard deviations are quite small; 
once again the exception is the parameter a. Scatter 
plots of the forecasted values versus the verification 
values indicated a linear relationship between the two 
sets, in several cases. Regression analyses to fit straight 
lines to the data confirmed this observation (Table 6 and 
Figures 19-30) . When the data was stratified according to 
the forecasts period, e.g., all 12 hour forecasts are 
treated as one group, and a separate regression analysis 
performed for each group the lienar relationship was 
accentuated (Table 7). 
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The overall conclusion is that the incorporation of 
the SEIS and VTP methodology within the NOGAPS model has 
improved the storm tracking capability of NOGAPS and the 
elliptic representation of storms provides a good means 
of providing synoptic level error statistics to the 
field forecasters. 

We propose the following topics for further study 
and research: 

1. Determine the most appropriate probability 
distributions to describe the probabilistic 
behavior of the forecast errors. The indications 
are that the Weibull family may provide a good fit 
to the absolute error data . 

2. Develop procedures to generate uncertainty contours/ 
confidence regions around the forecasted elliptic 
representations of a storm based on the probability 
distributions of the forecast errors. 

3. Examine more data to determine the functional 
relationship (if it exists) between the forecasted 
and verification values of the elliptic parameters. 
Different stratification schemes for the storm 
data such as by geographic regions and by climatic 
seasons could lead to the identification of sources 
of systematic errors and the means of remediation. 
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